{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quiz: the ResNet neural network\n",
    "==================\n",
    "\n",
    "In this notebook your goal is to implement a **ResNet-34** architecture. Moreover you have to implement an iterative method for generating the residual units. The network is similar to the ResNet-18 of the tutorial but the number of conolution is higher. Here you can find the table with a comparison between the two architectures:\n",
    "\n",
    "<p align=\"center\">\n",
    "<img src=\"../etc/img/resnet_type.png\" width=\"750\">\n",
    "</p>\n",
    "\n",
    "As you can see the number of filters and their size is equal in both architecture. Here instead of creating each residual unit by hand, you have to write an iterative method that does it for you. You have to train the model on the CIFAR-10, you can reuse the code of the tutorial for loading the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementation of ResNet-34\n",
    "--------------------------------\n",
    "\n",
    "Based on the code of the tutorial you have to write an iterative method for the creation of the ResNet-34. If the model is too big for running on your machine you are allowed to rewrite a ResNet-18 but using the iterative method. The **tag** `#Quiz` show you where to insert the additional code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_model_fn(features, labels, mode):\n",
    "    #Reshape the input    \n",
    "    x = tf.reshape(features, [-1, 32, 32, 3])\n",
    "   \n",
    "    #Feature maps: 32x32@64\n",
    "    with tf.variable_scope(\"unit_1\"):\n",
    "        conv1 = tf.layers.conv2d(inputs=x, filters=64, kernel_size=[3, 3], \n",
    "                                 padding=\"same\", use_bias=False, activation=None) #no ReLu activation here\n",
    "        conv1 = tf.layers.batch_normalization(conv1) #apply batch norm\n",
    "        conv1_out = tf.nn.relu(conv1) #ReLu after the batch norm\n",
    "        \n",
    "    #QUIZ: here you have to allocate the residual unit with a recursive solution\n",
    "    #You can try with a loop or with an external helper method.\n",
    "        \n",
    "    #Pooling and output\n",
    "    with tf.variable_scope(\"unit_6\"):\n",
    "        pool = tf.layers.average_pooling2d(inputs=conv5_out, pool_size=[2, 2], strides=1,\n",
    "                                           padding=\"same\")\n",
    "        pool_flat = tf.reshape(pool, [-1, 4 * 4 * 512])\n",
    "        logits = tf.layers.dense(inputs=pool_flat, units=10)\n",
    "    \n",
    "    #PREDICT mode\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        predictions = {\"classes\": tf.argmax(input=logits, axis=1),\n",
    "                       \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\"),\n",
    "                       \"logits\": logits}\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "    #TRAIN mode\n",
    "    elif mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        loss = tf.losses.softmax_cross_entropy(onehot_labels=labels, logits=logits)\n",
    "        #optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate=0.001)\n",
    "        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())\n",
    "        accuracy = tf.metrics.accuracy(labels=tf.argmax(labels, axis=1), predictions=tf.argmax(logits, axis=1))\n",
    "        tf.summary.scalar('accuracy', accuracy[1]) #<-- accuracy[1] to grab the value\n",
    "        #tf.summary.image(\"input_features\", tf.reshape(features, [-1, 32, 32, 3]), max_outputs=4)\n",
    "        #tf.summary.image(\"c1_k1_feature_maps\", tf.reshape(conv1[:, :, :, 0], [-1, 32, 32, 1]), max_outputs=4)\n",
    "        logging_hook = tf.train.LoggingTensorHook({\"accuracy\" : accuracy[1]}, every_n_iter=10)\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op, training_hooks =[logging_hook])\n",
    "    #EVAL mode\n",
    "    elif mode == tf.estimator.ModeKeys.EVAL:\n",
    "        loss = tf.losses.softmax_cross_entropy(onehot_labels=labels, logits=logits)\n",
    "        accuracy = tf.metrics.accuracy(labels=tf.argmax(labels, axis=1), predictions=tf.argmax(logits, axis=1))\n",
    "        eval_metric = {\"accuracy\": accuracy}\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resnet = tf.estimator.Estimator(model_fn=my_model_fn, model_dir=\"./tf_resnet_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model\n",
    "---------------------\n",
    "\n",
    "Here you have to reuse the code of the tutorial to load the dataset and train the model. The training can be considerably slower and it is possible that your machine cannot manage the network. If this is the case you can consider to downgrade the ResNet using less units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_input_fn():\n",
    "    #Quiz: rewrite the input function for the training\n",
    "    \n",
    "    \n",
    "    return batch_features, batch_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resnet.train(input_fn=my_input_fn, steps=4000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing the model\n",
    "--------------------\n",
    "\n",
    "Here you have to use again the code of the tutorial in order to test the model. You should notice an **improvement** in the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_eval_input_fn():\n",
    "    #Quiz: rewrite the input function for the test\n",
    "    \n",
    "    return batch_features, batch_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "resnet.evaluate(input_fn=my_eval_input_fn, steps=1000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
